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Knowledge  Point  1 :  Achieved  when  a  sound  business  case  is 
made,  matching  the  customer’s  requirements  with  the 
developers  resources  in  terms  of  knowledge,  time,  money 
and  management  capacity. 


Knowledge  Point  2:  Achieved  when  a  program  determines  that 
a  product’s  design  is  stable — that  is,  it  will  meet  customer 
requirements,  as  well  as  cost,  schedule,  and  reliability 
targets. 


Knowledge  Point  3:  Achieved  when  it  has  been  demonstrated 
that  the  company  can  manufacture  the  product  within  cost, 
schedule,  and  quality  targets. 
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Program  Launch 


Resources 

match 

requirements 


Design 
performs  as 
expected 


Production  can  meet 
cost,  schedule,  and 
quality  targets 


Knowledge  Point  - 1 


Knowledge  Point  -  2 


Knowledge  Point  -  3 
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Knowledge  Point  1  is  NOT  Just  TRLs 


•  Time  &  Money: 

•  Cost  and  schedule  estimates  based  on  knowledge  gained 
from  preliminary  design  and  systems  engineering 

•  Cycle  times  should  be  short  and  supported  by 
evolutionary  acquisition 

•  Cost  should  be  informed  by  risk  and  uncertainty  analyses 
and  presented  as  a  range  and  not  just  a  point  estimate 


•  Requirements: 

•  Informed  by  Systems  Engineering 

•  Clearly  Defined  and  Understood 

•  Stable 
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•  Time  and  Money: 

•  Business  case  cost  and  schedule  estimates  are  not  well 
informed 

•  Long  cycle  times  make  it  difficult  to  accurately  predict 
delivery  or  total  cost 

•  Baseline  cost  estimates  presented  at  a  “50%”  confidence 
level  without  risk  and  uncertainty  analysis 

•  Characterized  by  a  mindset  that  says  “we  can  fix  it  in  the 
next  FYDP” 


•  Requirements: 

•  Not  well  defined  or  understood,  e.g.  FCS,  JTRS 

•  Requirements  creep  is  the  norm,  e.g.  Global  Hawk 

•  Long  cycle  times  provide  opportunity  for  requirements  creep 
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Past  GAO  Recommendations  to  DOD  to 
Improve  Outcomes 


•  Implement  a  disciplined,  knowledge-based  review  process  prior  to  starting 
product  development. 

•  Require  increasingly  precise  cost,  schedule,  and  performance  information 
that  meets  specified  levels  of  confidence  and  allowable  deviations  at  each 
decision  point  prior  to  initiating  product  development. 


•  Bring  knowledge  about  resources  (time,  money,  and  technologies)  and 
requirements  together  early  to  inform  trades  and  manage  the  portfolio. 


•  Assign  and  empower  a  single  point  of  accountability  to  ensure  the  success 
and  balance  of  the  entire  acquisition  portfolio. 


•  Require  that  technologies  demonstrate  a  high  readiness  level — TRL  7 — 
before  Milestone  B  of  a  major  acquisition. 
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Despite  Constructive  Policy  Changes, 
Implementation  is  Still  A  Challenge 


•  DOD  5000  policy  says  most  of  the  right  things  about 

separating  technology  development  from  system  development 

•  Calls  for  technology  maturity  to  TRL  6  (relevant  environment) 

•  Calls  for  evolutionary  approach  as  a  check  on  requirements. 

•  Short  development  cycle  times  (5  years  or  less) 


•  However, 

•  Best  practice  standard  is  TRL  7  (operational  environment) 

•  Most  individual  programs  do  not  even  abide  by  policy 

•  Many  programs  fall  outside:  satellites,  MDA,  ships 

•  Those  within  are  unique:  e.g.,  FCS,  JSF 

•  Preference  is  still  for  revolutionary,  not  evolutionary 

•  Knowledge  gaps  and  optimistic  estimates  at  MS  B  are  the  norm 
and  are  reinforced  with  approval  and  funding 
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Technology  Maturity  for  18  Programs  Initiated 
the  Revised  Acquisition  Policy 


Note:  Maturity  measured  against  the  DOD  standard  of  TRL  6 
Source:  GAO-06-368 
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Percentage  of  Programs  that  Achieved  TRL  7 
at  Key  Junctures 


Development  start  DOD  design  Production 

review  decision 

Note:  If  DOD’s  standard  of  TRL  6  is  used  32%  of  the  programs  entered  development  with  all  their  technologies  mature. 
Source:  GAO-07-406SP 
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Best  Practices  for  Technology  Transition 
Report  Objectives 


•  Identify  techniques  used  by  leading  private  companies  to 
transition  mature  technologies  to  product  lines  by  product 
launch. 


•  Assess  practices  used  by  the  military  services  to  transition 
technology. 


•  Determine  potential  technology  transition  practices  DOD  could 
use  to  improve  its  weapon  systems  outcomes. 
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•  Merge  technology  development  and  product  development 
activities  prior  to  product  launch. 

•  Have  strong  strategic  planning  to  prioritize  technology  needs 
and  a  structured  technology  development  process. 

•  Use  3  tools  to  support  technology  transition: 

•  Relationship  managers 

•  Technology  Transition  Agreements 

•  Metrics 
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Private  Industry  Findings  -  Merge 
Technology  and  Product  Development 


General  Flow  of  Process  Leading  Up  to  Technology  Transition 


Strategic 

1  i 

Technology  development 

planning 

Explore 

Develop 

TRANSITION 

Product  development 


Program 

start 


Souce:  GAO  analysis  and  pssantaUcn  ot  leaJlng  ctonpartse' 

•  Hybrid  phase  used  to  merge  technology  development  and 
product  development  activities  prior  to  product  launch. 


•  Responsibilities  for  managing  and  funding  technology 
development  gradually  shift  from  labs  to  product  line  during  this 
phase. 
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Private  Industry  Findings  -  Precursors  to 
Smooth  Transition 


•  Strong  strategic  planning  processes  used  to  identify  and 
react  to  market  needs  quickly. 

•  Structured,  gated  technology  development  process. 


TECHNOLOGY  DEVELOPMENT  GATES 

Explore 

Technology  ideas  and  concepts 
are  being  explored 

flsv 

A 

Develop 

Technology  development  activities 
are  underway 

'iew  Re\> 

i.  A 

Technology  transition 

Technology  is  ready  to  transition 
from  lab  to  product  line  team 

low 

k 

Deliverables 

•  Technology  is  consistent  with  overall 
business  strategy 

•  Technology  is  promising  and  is  likely  to 
meet  needs  for  potential  product  lines 

•  Lab  identifies  potential  products  where 
technology  can  be  used 

•  Key  cost,  benefit,  risk,  marketing, 
manufacturing,  and  life  cycle  manage¬ 
ment  issues  are  identified 

•  Scalability  approaches  are  identified 

•  Technologies  considered  to  be 
intellectual  property  are  identified 

Deliverables 

•  Technology  is  consistent  with  techno¬ 
logy  strategy  and  other  relevant 
strategies 

•  Labs  have  high  degree  of  confidence 
the  technology  will  work 

•  Product  line  team  agrees  that  the 
technology  will  meet  its  needs 

•  Technical  requirements  are  identified 

•  Cost,  benefits,  and  risks  are  quantified 

•  Scalability  approach  is  selected 

•  Strategies  for  addressing  intellectual 
property  rights  are  selected 

Deliverables 

«  Technology  project  complies  with 
technology  strategy 

•  Technology  is  sound 

•  Technology  meets  product  require¬ 
ments 

•  Cost,  benefit,  and  risks  are  well 
understood 

•  Technology  can  be  scaled  to  a 
magnitude  appropriate  for  practical 
application 

•  Product  line  team  agrees  technology  is 
ready 

•  Intellectual  property  rights  methods 
have  been  pursued 

•  Technology  is  demonstrated  in  an 
operational  environment 

•  Technical  documentation  is  ready  to  be 
given  to  product  line  team 

Private  Industry  Findings  -  Use  of 
Relationship  Managers 
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Relationship  managers  from  labs  and  product  lines  serve 
as  a  communication  link  between  the  two  communities 
and  work  out  transition  issues. 


LAB  COMMUNITY 


Executive  managers 

1 

Liaisons 

1 

Technologists 


4 


4 


Sourc-a:  Motorola;  GAD  QanayslB  and  fr&sanlaflcn;i. 
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PRODUCT  LINE 
COMMUNITY 


Executive  managers 

I 

Liaisons 


Technologists 
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Private  Industry  Findings  -  Use  of  Technology 
Transition  Agreements 


•  Technology  transition  agreements  document  decisions  made 
between  labs  and  product  lines: 

•  Contain  specific  quantifiable  cost,  schedule,  performance, 
and  manufacturability  metrics  the  labs  must  demonstrate 
before  product  line  acceptance. 

•  Feasibility,  relevancy,  and  application  of  each  technology 
are  assessed  in  order  to  identify  potential  barriers  to 
transition. 

•  Identify  lab  and  product  line  funding  commitments. 

•  May  include  loaning  key  lab  technologists  to  the  product 
line. 
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Private  Industry  Findings  -  Use  of  Project  and 
Process  Metrics 


•  Project  metrics  used  to  assess  the  status  of  technology 
development  and  whether  the  technology  meets  product 
needs 

•  Size,  weight,  power,  and  reliability,  as  well  as 
nonrecurring  development  and/or  manufacturing  costs. 


•  Process  metrics  provide  information  on  the  status, 
timeliness  and  impact  of  technology  development  efforts. 

•  Return  on  investment,  cycle  time,  technology  yield, 
number  of  technologies  commercialized,  customer  survey 
results. 
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Notional  Boeing  Technology  Maturity 
Scorecard  for  a  Hypothetical  Technology 


Criteria  for  readiness 


1 .  Consistency  with 
strategy 


Technology  development 
Discovery  Feasibility  Practicality 


2.  Technical  validity 


3.  Cost,  benefit,  risk 
assessment 

4.  Competitive  technology 
assessment 

Technology  transition 

5.  Scalability 


6.  Collateral  impact 


7.  People  and  organization 
readiness 

8.  Product  line 
endorsement 
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Source:  GAO  analysis  based  on  The  Boeing  Company’s  scorecard. 
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•  DOD  does  not  adequately  prioritize  the  technologies  that  are 
most  critical  to  acquisition  programs. 

•  DOD  does  not  merge  S&T  and  product  line  activities  prior  to 
product  launch;  Transition  often  occurs  at  product  launch 
irrespective  of  whether  technologies  are  mature. 

•  New  tools  to  support  transition  are  being  used,  but: 

•  Not  as  comprehensive  as  industry  best  practices. 

•  Use  is  not  widespread. 
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DOD  Findings  -  DSB  Report 
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Immature  technologies  and  manufacturing  challenges  have  a  significant  Impact  on 
DOD's  ability  to  rapidly  and  affordably  transition  technology  to  the  war  fighter," 

"S&T  program  managers  often  believe  that  affordability  and  manufacturing  issues  are  not 
relevant  concerns  in  6.3  programs,  focusing  instead  on  fabrication  of  test  and  evaluation 
and  prototype  articles.  But  this  line  of  thinking  leads  to  higher  costs  later  in  a  prog  ram , 
when  manufacturing  concerns  are  addressed  after  technical  designs  are  considered 
'ready/" 

In  order  to  achieve  the  objective  of  lower  cost  equipment,  manufacturing  concerns  must 
be  addressed  earlier  in  the  program  life  cycle.  Production  and  support  costs  need  to 
become  a  component  of  key  technical  design  requirements,  before  the  final  stages  of 
development  when  technologies  are  released  for  prototyping " 

Sruroa:  Defense  Science  Board  study  (dale):  GAO(Ffesentedon  andenelyafe). 
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DOD  Findings  -  Underdeveloped  Technology 
Prioritization  and  Development  Processes 


DOD  is  not  well  positioned  to  develop  and  mature  needed 
technologies  on  time. 

•  Strategic  planning  process  does  not  consistently  prioritize 
technologies  most  critical  to  acquisition  programs. 


•  Military  services  have  established  S&T  boards  to  select 
and  oversee  new  technology  projects,  which  increases 
visibility  for  some  technologies,  but  the  scope  varies 
across  military  services. 
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DOD  Findings  -  Technology  and  Product 
Development  are  not  Effectively  Aligned 

DOD  does  not  have  a  structured,  gated  S&T  technology 
development  process  with  deliverables  to  guide  investments. 


Technologies  are  Immature 
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Acquisition  community  is  responsible  for  technology  development  AND  product  development 


Technology  development 


Technology  development 
Responsibility:  Lab  community 


> 


Product  development 


Soiree:  DOD  (dots) .  OAO  isnafyste  am)  ptaoentaUroi. 


•  S&T  and  acquisition  communities  do  not  communicate  well  and 
are  not  aligned  in  a  way  to  effectively  meet  priorities,  resulting  in: 

•  Irrelevant  technologies  advancing  to  final  stages  of  lab  development  without 
commitment  to  field  the  technologies. 

•  Technology  not  being  ready  to  transition  when  needed. 

•  Acquisition  not  being  prepared  to  take  over  funding  responsibilities. 
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DOD  Findings  -  Tools  to  Support  Technology 
Transition  are  Underutilized 


•Relationship  Managers 

•  Generally  used  to  market  lab  technology;  not  as  a  communication 
tool  to  assist  in  technology  transition. 

•Technology  Transition  Agreements 

•  Use  and  coverage  vary  greatly  among  service  S&T  programs 

•  Agreements  contain  some  of  the  same  elements  seen  in  industry, 
but  typically  do  not  require  the  technology  developer  to 
demonstrate  cost  metrics. 

•  Tool  used  mainly  by  labs;  not  valued  by  acquisition  community  as 
highly. 

•Metrics 

•  Few  metrics  used  to  gauge  the  impact  of  investments  or  the 
effectiveness  of  processes  used  to  develop  and  transition 
technologies. 
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DOD  Findings:  Some  Promising  Initiatives  to 
Aid  Transition 


•  Advanced  Concept/Joint  Concept  Technology  Demonstration 

(ACTD/JCTD)  --  Goal  is  to  get  technologies  that  meet  critical  needs 
to  users  faster  and  at  lower  cost,  refine  the  the  selection  process  to 
better  match  user  priorities,  and  provide  more  funding  in  early 
stages  of  demonstration. 

•  Manufacturing  Technology  Program  --  Aimed  at  quickly 
identifying  and  solving  technology  transition  problems;  focusing  on 
affordable,  low-risk  development  and  production 

•  Foreign  Comparative  Testing  &  Technology  Transition  Initiative 

--  FCT  identifies,  evaluates,  and  procures  technologies  developed 
by  other  countries.  TTI  speeds  transition  of  DOD  lab  developed 
technologies  to  acquisition  programs. 
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•  Develop  a  gated  process  for  developing  and  transitioning  technologies  that 
establishes  a  transition  phase  and  defines  activities  that  should  occur 
during  this  phase. 

•  Set  aside  a  portion  of  advanced  component  development  and  prototype 
funds  for  the  S&T  to  manage  the  transition  of  technologies  to  acquisition 
programs. 

•  Expand  the  use  of  technology  transition  agreements  to  applied  and 
advanced  development  projects. 

•  Include  additional  metrics  in  technology  transition  agreements. 

•  Expand  the  use  of  relationship  managers  and  define  responsibilities. 

•  Adopt  additional  process-oriented  metrics  to  measure  the  effectiveness  of 
S&T  processes  and  the  impact  of  S&T  investments. 
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